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Abstract: Motivated by applications in telecommunications, computer science and physics, 

we consider a discrete-time Markov process with restart in the Borel state space. At each step 
the process either with a positive probability restarts from a given distribution, or with the com¬ 
plementary probability continues according to a Markov transition kernel. The main focus of the 
present work is the expectation of the hitting time (to a given target set) of the process with 
restart, for which we obtain the explicit formula. Observing that the process with restart is posi¬ 
tive Harris recurrent, we obtain the expression of its unique invariant probability. Then we show 
the equivalence of the following statements: (a) the boundedness (with respect to the initial state) 
of the expectation of the hitting time; (b) the finiteness of the expectation of the hitting time for 
almost all the initial states with respect to the unique invariant probability; and (c) the target set 
is of positive measure with respect to the invariant probability. We illustrate our results with two 
examples in uncountable and countable state spaces. 
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Le Temps de Premier Passage dans les Processus de Markov 

avec Redemarrage: 

Raison du Travail de Sisyphe 

Resume : Motive par diverses applications provenant de telecommunications, informatique 

et de la physique, nous considerons un processus generate de Markov dans l’espace de Borel avec 
une possibility de redemarrage. A chaque etape, avec une probability le processus redemarre 
a partir d’une distribution donnee et avec la probability complementaire le processus continue 
revolution selon une noyau de Markov. Nous etudions l’esperance du temps de premier passage 
a 1’ensemble donne. Nous obtenons une formule explicite pour l’esperance du temps de premier 
passage et demontrons que le processus avec redemarrage est Harris positif recurrent. Ensuite, 
nous etablissons que les assertions suivantes sont equivalentes : (a) le fait d’etre limite (par 
rapport a l’etat initiate) de l’esperance du temps de premier passage; (b) la finitude de Fesperance 
du temps de premier passage pour presque tous les etats initiaux par rapport a la probability 
invariante unique; et, (c) l’ensemble cible est de mesure positive par rapport a la probability 
invariante. Enfin, nous illustrons nos resultats theoriques avec deux exemples dans les espaces 
denombrables et non denombrables. 

Mots-cles : Processus de Markov en Temps Discret, L’Esperance du Temps de Premier Passage 
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1 Introduction 


We study a discrete-time Markov process in the Borel state space with restart. At each step 
the process either with the positive probability p restarts from a given distribution, or with 
the complementary probability 1 — p continues according to a Markov transition kernel. Such 
processes have many applications in telecommunications, computer science and physics. Let us 
cite just a few. Both TCP (Transmission Control Protocol) and HTTP (Hypertext Transfer 
Protocol) can be viewed as protocols restarting from time to time, Krishnamurthy and Rexford 
m, Maurer and Huberman m■ The PageRank algorithm, Brin and Page [5], in information 
retrieval models the behaviour of an Internet user surfing the web and restarting from a new 
topic from time to time. Markov processes with restart are useful for the analysis of replace 
and restart types protocols in computer reliability, Asmussen et al. (2008) 0, Asmussen et al. 
(2014) Kulkarni et al. [14 . The restart policy is also used to speedup the Las Vegas type 
randomized algorithms Alt et al. [I], Luby et al. mi- Finally, human and animal mobility 
patterns can be modeled by Markov processes that restart from some locations Gonzalez et al. 

EH. Walsh et al. [IS] . 

We observe that the process with restart exhibits several desirable self-stabilizing properties. 
For example, it is always positive Harris recurrent with the unique invariant probability (see 
Corollary 2.11. Moreover, for a fixed target set, when the (expected) hitting time of the original 
process without restart is infinite (recalling the Polya theorem for random walks), the one of 
the process with restart could be still finite. The present work elaborates on this point. As a 
main result we show (see Theorem 2.31 the equivalence between the following statements, some 
of which are otherwise superficially stronger than the others. 


(a) The expected hitting time of the process with restart is finite for almost all the initial states 
with respect to the invariant probability. 

(b) The expected hitting time of the process with restart is (uniformly) bounded with respect 
to all the initial states. 


(c) The target set is of positive measure with respect to the invariant probability. 


This characterization result is based on the obtained explicit and simple formula of the expected 
hitting time of the process with restart in terms of the expected discounted hitting time of 
the process without restart; see Theorem |2.2[ whereas the formula itself can be useful in the 
optimization of the expected hitting time of the process with restart with respect to the restart 
probability; see Section [3] for examples. 

Let us mention some related work to the present one in the current literature. The continuous¬ 
time Markov process with restart was considered in Avrachenkov et al. |!5j. According to Theorem 
2.2 in |5], the continuous-time Markov process with restart is positive Harris recurrent in case 
the original process is honest. At the same time, the process with restart is not positive Harris 
recurrent if the original process is not honest (i.e., the transition kernel is substochastic; in case 
the state space is countable, that means the accumulation of jumps). The objective of [5] does not 
lie in the expected hitting time, but in the representation of the transition probability function of 
the (continuous-time) process with restart in terms of the one of the original (continuous-time) 
process without restart. This is trivial in the present discrete-time setup. Here our focus is on 
the characterization of the expected hitting time. We also would like to mention the two works 
Dumitriu et al. 0, Janson and Peres m , dealing with the control theoretic formulation, where 
the controller decides (dynamically) whether it is beneficial or not to perform a restart at the 
current state. That line of research can be considered complementary to ours. 
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The rest of this paper is organized as follows. The description of the process with restart and 
the main statements are presented in Section [2] which are illustrated by two examples in Section 
[3j The paper is finished with a conclusion in Section [4] 


2 Main statements 


Let us introduce the model formally. Let E be a nonempty Borel space endowed with its Borel 
cr-algebra 13(E). Consider a discrete-time Markov chain X = {Xt, t = 0,1,...} in the Borel 
state space E with the transition probability function P(x,dy ) being defined by 


P(x,r) :=p V (T) + (l-p)P{x,T), 


for each T £ 13(E), where p £ (0,1) and P(x, dy) is a transition probability function, and v is a 
probability measure on 13(E). Let X := {X t ,t = 0,1,... } denote the Markov chain corresponding 
to the transition probability P(x, dy). We assume the two processes X and X are defined on 
the common probability space (f2, T, P); if we emphasize that the initial state is x £ E, then E x 
denotes the corresponding expectation operator, and the notation P x is similarly understood. 

The process X is understood as the modified version of the process X, and is obtained by 
restarting (independently of anything else) the process X after each transition with probability 
p £ (0,1) and the distribution of the state after each restart being given by v\ whereas if there 
is no restart after the transition (with probability 1 — p), the distribution of the post-transition 
state is P(x,dy) (given that the current state is x). 

The following notation is used throughout this paper. Let P t (x, dy), t = 0,1,..., be defined 
iteratively as follows; for each T £ 13(E), 


P°(x, r) 

P t+1 (x,T) 


I{x £ T}, 

[ P t (x, dy)P(y, T) 
J E 


Theorem 2.1 The process X has an invariant probability measure q(dy) given by 



oo 


(1) 


for each T £ 13(E). 


Proof. Clearly, q(dy) is a probability measure. We now only need to show that q(dy) is invariant 
by verifying that 
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for each F £ 13(E) as follows. Indeed, 

[ q(dx)P(x, r) = [ q(dx)(pu(T) + (l-p)P(x,T)) 

J E J E 

n OO n 

= pv(T) + (l-p) P(x,T)^2p(l- P y P t (y, dx)u(dy) 
J E t=0 JE 

OO r. 

= M r ) +^2p(l-p) t+1 / P t+l (y,T)v(dy) 
t=o •* E 

OO ,, 

= Pv(T) +Vp(l -p) T / P T (y,T)v(dy)-pv(T) 

JE 


T — 0 


= q( r). 

The proof is completed. 


As stated in Corollary 
of the process X. 

The process X is ^-irreducible, i.e., for each set T £ B(E) satisfying i/(T) > 0, it holds that 
P x (tt < oo) > 0 for each x £ E\ see p.87 of Meyn and Tweedie [I7j. Here and below 

r r := inf{t = 1, 2, • • • : X t £T}. (2) 

As usual, inf0 := oo. Such a measure v is called an irreducibility measure. It follows from the 
proof of Proposition 4.2.2 of Meyn and Tweedie m that the measure q(dy) given in Theorem 
|2.1| is actually the maximal irreducibility measure; that means, any irreducibility measure is 
absolutely continuous with respect to q(dy). Throughout this work, we adopt the definitions 
related to a discrete-time Markov process such as the Harris recurrence and so on all from the 
standard reference Meyn and Tweedie CZI- 

In this paper we are primarily interested in the expected hitting time of the process X to a 
given set H £ 13(E), defined by 

VH ■= inf{i = 0,1, • • • :X t £ H}. 


2.1 below, q(dy) is actually the unique invariant probability measure 


For the future reference, we put 

H P°(x, E) := I{x £ E\H}. 

Denote 


and let 


V(x) := E x [r) H \ 


hP(x,Y) 


P(x,T ), T£H, 
0, r c H 


be the taboo transition kernel with respect to the set H. Then, one can write 


V(x) = l+p [ V(y)u(dy) + (1 - p) [ V(y) H P(x,dy), V x £ E\H, 
J E JE 

V(x) = 0, V x £ H. 


(3) 


(4) 
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Furthermore, it is well known that the function V ( x ) defined by ([3]) is the minimal nonnegative 
(measurable) solution to equation Q, and can be obtained by iterations 

y (n+1) (:r) = 1 +p j V {n \y)v(dy) + (1 - p) [ V {n \y) H P(x,dy), n = 0,1,... 

J E J E 

with V( n \x) = 0 if x £ H, and V^°\x) = 0; c.f. e.g., Proposition 9.10 of Bertsekas and 
Shreve [7]. 

One can actually obtain the minimal nonnegative solution to Q in the explicit form. 

Theorem 2.2 (a) The minimal nonnegative solution to w is given by the following explicit 
form 


V{x) = Vi(x)^2(pj Vi(y)v(dy)j , \/x€E\H , 


V(x) = 0, V x G H, 
where the function V\ is given by 


Vi(x) := ^(1 - p) t H P t {x,E), \/xGE\H ; 

4=0 

V\(x) := 0, \/xGH. 

It coincides with the unique bounded solution to the equation 


(5) 


( 6 ) 


V\(x) = 1 + (1 — p) f Vi(y) H P{x,dy), \/xGE\H- 
J e 

V\ (x) = 0, \/xGH. (7) 


(b) If q(H) > 0, then 


Vlx) = _^_ 

1 -pSe v i(yM d y) 

V(x) =o, V x G H. 


V xeE\H, 


( 8 ) 


Proof, (a) Observe that the function V\ given by © represents the expected total discounted 
time up to the first hitting of the process X at the set H given the initial state x and the discount 
factor 1 —p. It thus follows from the standard result about the discounted dynamic programming 
with a bounded reward that the function V\ is the unique bounded solution to equation Q; see 
e.g., Theorem 8.3.6 of Hernandez-Lerma and Lasserre m 

Now by multiplying both sides of the equation |7]) by the expression 

Vi(y)i'{dy^j 

for all x £ E\H, it can be directly verified that the function V defined in terms of V\ by ^ is a 
nonnegative solution to (©• We show that it is indeed the minimal nonnegative solution to @ 
as follows. 

Let U ( x ) > 0 be an arbitrarily fixed nonnegative solution to Q . It will be shown by induction 
that 



U(x)>V i(z)E 

t=0 



Vi(y)v(dy) 


V n = 0,1,.... 


(9) 
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The case when x £ H is trivial. 

Let x £ E \ H be arbitrarily fixed. It follows from 0 that 

U(x) > 1 + (1 -p) J H P{x,dy) = ^(1 - pf ^ J H P(x,dy)^j > 1. 


If for some n > 1 


then by Q, 


u ( x )>pi- P y (J e hpM)) , 

U{x ) > 1 + (1 -p) [ U(y) H P(x,dy) 

J E 

> 1 + (1 ~P) J e ^X](l ~ p ^ (yj E hP ( x ' dy ^j j hP(x, dy) 

n+1 / - \t 

= ~P^ \J E HP ^ X ' dy ^) ’ 

and so (111 holds for all n > 0 and thus by |6| 

U{x) > Vi(x). 


( 10 ) 


( 11 ) 


Consequently, ([9| holds when n = 0. 

Suppose ^ holds for n, and consider the case of n + 1. Then from Q, 

U(x) = l+p [ U(y)v{dy) + (1 - p) [ U{y) H P{x,dy ) 

Je Je 

Tl~\~ 1 / n \ t /» 

> ^2\P J E Vi(y)v(dy)J + (1 - p) U(y) H P(x, dy), (12) 

where the inequality follows from the inductive supposition. Define the function W on E by 

U(z) 


W(z) = 


E ?= 0 (pf E v i( y ) l/ ( dy )) 


Vzefi. 


Then by (12 I, 


W(x) > 1 + (1 — p) f H P(x,dy) = '^2(l-p) t ( f H P(x, dy)\ ; 

J E t—0 E / 


c.f. (10). Now, based on (121, a similar reasoning by induction as to the verification of (111 for 
all n > 0 shows that 

w ( x ) > X](! ~pY (j E HP ( x >dy)j , V k = 0,1,..., 
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and thus W{x) > V\(x). This means 


n +1 , ,. 

U(x) > V\(x) ^ [p / Vl( y)iy(dy) 
t—O V Je 


Thus by induction, ([9]) holds for all n > 0, and thus 

U(x) > V{x) 

by© , as desired. 

(b) If q{H) > 0, then there exists some T > 0 such that f E P T (y , H)v(dy) > 0, meaning that 
there exists some T' <T such that f E hP t ' {x, E)v(dx) < 1. Thus, 


p OO 

0<p X](l -P) t H P t {y,E)v{dy) 
•* E t=o 


< 1, 


and so the geometric series (pIe {y) l '{dy)) t converges. The statement follows. □ 

Corollary 2.1 The process X is positive Harris recurrent with the unique invariant probability 


measure q(dy) given in Theorem 2.1 


Proof. Recall the discussion below Theorem 2.1 that q{dy) is the maximal irreducibility measure. 
For each measurable subset T of E satisfying q(T) > 0, 

E x [rr] = E x [t t \X x G T]P,(X, G T) + E x [r r|Xr ^ T }P x {X 1 £ T) 

< 1+ / E y \r]r\P{x, dy) < oo 

Je\t 

for each igT, where rr is defined by ©, and the last inequality follows from the boundedness 
of V ; see Theorem 2.2 b). Thus, P x {tt < oo) = 1 for each iGT. By Proposition 9.1.1 of Meyn 
and Tweedie m , t his m eans the chain X is Harris recurrent. Now by p.231 of Meyn and Tweedie 


m and Theorem |2.l| X is positive Harris recurrent. The fact that the invariant probability 


measure q is the unique (cr-finite) invariant measure follows from Theorem 10.4.4 of Meyn and 
Tweedie nzi. ' □ 


The next corollary is immediate. 

Corollary 2.2 If q(H ) > 0, then both V\(x) and V(x ) are bounded with respect to the state 
x G E. In particular, we have 

V\(x) < -. 

P 


We also have the following chain of equivalent statements, some of which are otherwise su¬ 
perficially stronger than the others. 

Theorem 2.3 The following statements are equivalent. 

(a) q{H) >0. 

(b) V{x) = E x [t/h\ < oo for each x G E. 

(c) V(x) = E x [t]h] < oo for almost all x G E with respect to q(dy). 

(d) swp xeE V(x) = sup x&E E x [r) H ] < oo. 
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The proof of this theorem follows from the next lemma. 


Lemma 2.1 Statements (a) and (c) in Theorem 2.3 are equivalent. 

Proof. Statement (a) implies statement (c) by Theorem |2.2[ b). It remains to show that statement 
(c) implies statement (a). To this end, suppose for contradiction that statement (a) does not 
hold, i.e., q(H) = 0. Then it follows from ([l]) that 


p OO 

/ ^2 pt (y^ H M d y) = °- 

J E 4. — n 


E t =o 

Thus, there exists a measurable subset T of E \ H such that 

*(T) = 1 

and 


(13) 


Px[vh = °°] = i v x g r, (14) 

where 

Th := inf{£ = 0,1, ■ ■ • : X t G H}. (15) 

Let x G T be fixed. Then by |6| and ( fTlj ) 





OO 

Vi(x) = E x 

E (»-»>• 

t=0 

— E x 

E a-?)' 

0 


Since x G T is arbitrarily fixed, and i/(T) = 1 by (131, we see from ([5]) and the fact that Vi(y) > 1 
if y G E\H that V(y) = oo for each y G E\ H. Since q{E \ H) = 1, we see that statement (c) 
does not hold. □ 


Proof of Theorem 2.3 From Theorem |2.2[ b), we see that (a) implies (b), which implies (c). 
From Lemma 2.1 (c) implies (a). Clearly, (d) implies (b). Finally, (a) implies (d) because V\{x) 
is bounded; see Corollary |2.2[ □ 


Next let us consider the dependance of V(x) on the restart probability p for each fixed x G E. 
When we emphasize the dependance on p , we explicitly write V(x,p) and V\{x,p). In the above, 
p G (0,1) was fixed. Now we formally put 

OO 

V(®,0) := Y^mP'^tE) G [1,oo],V xGE\H- 
t =o 

V(a;,0) := 0,V x G H, 

which represents the expected hitting time of the process without restart, and 

V{x,l) := G [l,oo],V x G E\H; 

V(x,l) := 0,V x G H, 

which represents the expected hitting time of the process that restarts with full probability at 
each transition. Here and below, § := oo for any c > 0. As usual, the continuity of V(x,p) at 
p = a means limp^a V(x,p) = V(x, a) G [—oo, oo]. 

The next fact will be frequently referred to, and is a well known consequence of the dominated 
convergence theorem; we recall it here for completeness. 
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Lemma 2.2 Let (f be a measure space. Suppose the real valued function f(u>,p) is con¬ 
tinuous in p £ [a, b] C K. for each fixed lo £ fl; is differentiable in p £ (a, b) for each fixed ui £ LI; 
and is integrable in u> for each fixed p £ (a, b) C R. If there is an integrable function g on LI such 
that for each u> £ LI, |^(p,w)| < g(u) for each p £ (a,b), then f (u, p) p(du)) is differentiable 
in p £ (a, b), and 


d 

dp , 


f(u],p)p(du) = 


df 

dp 


(u,p)n(du). 


Theorem 2.4 Suppose 


p OO 

/ Y, pt (y’ H >(dy)> 0 . 

t =0 


(16) 


Then the function V[x,p) is infinitely many times differentiable in p £ (0,1) and is continuous 
in p £ [0,1], As a consequence, the problem 


V{x,p) 


mm 

pe[o,i] 


(17) 


is solvable. 


Remark 2.1 The condition (16) in the above statement is equivalent to q(H) > 0 for some and 
then all p £ (0,1) by 0- 


Proof of Theorem \2.f\ If x £ H, the statement holds trivially since V(x, p) = 0 for each p £ [0,1]. 
Consider now x £ E\H. Applying Lemma 2.2 one can show that V\{x,p) (as given by ©) and 
(1 — p f E Vi(y,p)v(dy))~ 1 are both infinitely many times differentiable in p £ (0,1). It follows 
from this and © that V(x,p) is infinitely many times differentiable in p £ (0,1). For the the 
continuity of V(x,p) at p = 0, it holds that 


1 ~P Vffy,p)v{dy) = 1 - p E y 


7 

r 



v{dy) = 1 - / E v 

1 - (1 -Pff” 

t =0 

J E 



'( dy ) 


E... 


(1 — p) VH i'(dy) —> 1 


as p —> 0 by the monotone convergence theorem; recall (151 for the definition of . It follows 
from this fact, and the monotone convergence theorem that 


lim V(x p) = _ lim p^o Vi{x,p) _ 

p^° ’ lim p _>o {1 -pf E V 1 (y,p)i/(dy)} 


= V(x,0) 


as desired. The continuity of the function V[x,p) at p = 1 can be similarly established. The last 
assertion is a well known fact; see e.g., Bertsekas and Shreve (7] . □ 


3 Examples 

3.1 Uni-directional random walk on the line 

Let E = M, H = [a, b] with a < b being two real numbers. The process X only moves to the 
right, and the increments of each of the transitions are i.i.d. exponential random variables with 
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the common mean ^ > 0. The restart probability is denoted as p £ (0,1) as usual, and the 
restart distribution v is arbitrary. Below by using Theorem |2.2| we provide the explicit formula 
for the expected hitting time at H of the restarted process X. (Clearly, if the initial state is 
outside H , then the expected hitting time of the process X at the set H is infinite.) 

We can give the following informal description of this example. There is a treasure hidden in 
the interval [a, b} and one tries to find the treasure. Once the searcher checks one point in the 
interval [a, &], he finds the treasure. The searcher has the means only to stop and to check points 
between the exponentially distributed steps. This models the cost of checking frequently. It is 
also natural to restart the search from some base. Intuitively, by restarting too frequently, the 
searcher spends most of the time near the base and does not explore the area sufficiently. On 
the other hand, restarting too seldom leads the search to very far locations where the searcher 
spends most of the time for nothing. Hence, intuitively there should be an optimal value for 
restarting probability. 

One can verify that in this example the unique bounded solution to Q is given by V\ (x ) = 0 
for each x £ [ a,b ], 

Vi(x) = - (18) 

P 

for each x > 6, 

VAx) = -- (1 - e-^ b ~ a) ) e~^ a ~ x)p (19) 

p p \ ) 


for each x < a. In fact, this can be conveniently established using the following probabilistic 
argument. Recall that V\(x) represents the expected total discounted time up to the hitting of 
the set H by the process X ; see ([6]). So for (181 one merely notes that with the initial state 


x > b, rjfl = oo, where r/)) is defined by (15). For (191, one can write for each x < a that 


V^x) = E x 


'vh-1 



'vh~ i 


£('- p)‘ 

II 

E x 


n X 

Vh 

O 

i! 



O 

Jl 



Now the expected hitting time of the restarted process X to the set H = [a, b] is given by 


1 1 — p 

p p 


(l - e-^ b - Q )) e -M(a-«)^ J2 [pj E ViivHdy)^ , 


V{x) = 

with the initial state x < a, and by 

v ( x ) = - ( p \ v i(yM d y) 
Pt^O V Je 


with the initial state x > b, recal l (f5j) . If the restart distribution v is not concentrated on (6, oo), 
then q{H) > 0, and by Theorem |2.2[b) we have 


V(x) = 


1 _ 1 -P ^ _ g -u(b-a)^ e ~n(a—x)p 

i-pf E v i(y)v( d y) 


with the initial state x < a and by 

V ^ p(i-pf E v i(y)v( d y))’ 
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for each x > b. In particular, if the process restarts from a single point r < a, the above 
expressions can be specified to 

1 - (1 - p)(l - e -^b-a)^ e -^a-x)p 
^ X - p( 1 - p)(l - e -n(b-a)^ e -iJ,(a-r)p ’ 

for the initial state x < a and to 


V{x) = 


1 _ e -n(b-a) J ^£>(1 — p'j e ~n(a.-r)p 


for each x > b. For the latter case (x > b), by standard analysis of derivatives, one can find the 
optimal value of the restart probability minimizing the expected hitting time of the process with 
restart in a closed form, as given by 


Popt 


2 + /r(a - r) + + p 2 (a - r) 2 


Now we can make several observations: the first somewhat interesting observation is that in the 
case when the initial state is to the right of the interval [a, b], the value of the optimal restart 
probability does not depend on the length of the interval but only on the average step size and 
on the restart position. The second observation is that when /i{a — r) is small, i.e., when either 
the average step size is large or the restart position is close to H , the optimal restart probability 
is close to 1/2. Thirdly, when /x(a —r) is large, the optimal restart probability is small and reads 


1 


Popt — 


1 + /r(a — r) 


1 


P(a~r) 


3.2 Random walk on the one dimensional lattice 

Let us consider a symmetric random walk on the one dimensional lattice which aims to hit 
H = {0} with restart at some node r. Assume without loss of generality that the restart state r 
is on the positive half-line, i.e., r > 0. 

From Theorem |2.2| we conclude that it is sufficient to solve the following equations 

Vi(k) = 1 + (k - 1) + V^k + 1)], k 0, 

14 ( 0 ) = 0 . 

Following the standard approach for solution of difference equations, we obtain 

Vi (k) = caq + -, 

P 

where aq < 1 is the minimal solution to the characteristic equation 

a= ^^[1 + a 2 ], 

and the constant c = comes from the condition 14(0) = 0. Consequently, 

1 -a\ 


V ( k ) = 

1 — pVi(r) pa\ 
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An elegant analysis can be done for the limiting case when the initial position k goes large, and 
hence we now minimize lim^oo V(k) = 1 /{poif), or equivalently, maximize pa\ with respect to 
p £ (0,1). This leads to the following equation for the optimal restart probability 


p da i 1 

a\ dp r 


( 20 ) 


Indeed, we note that 


p dai 

lim--— 

p->o ai dp 


= 0 , 


p da 1 

lim- - — = — oo, 

p->-i a\ dp 


and 


d_ ^ p_da __1_ 1 - (1 -p 2 )(l -p) 

dp\at dp ) ^1 - (1 - p) 2 (1 - p) 2 (l - (1 - p) 2 ) 


Thus, the left hand side of (201 is a monotone function decreasing from zero 
Consequently, the unique solution of equation (20 ) is the global minimizer 
equation (20 ) can be transformed to the polynomial equation 


to minus infinity, 
of 1 /(pa\). The 


^(1 -P) 2 (2 ~p) =P 


Consider the case of large r. This is a so-called case of singular perturbation, as the small 
parameter 1/r 2 is in front of the largest degree term, Avrachenkov et al. [3], Baumgartel [Sl¬ 
it is not difficult to see that for large values of r, the equation has one real root that can be 
expanded as 


_ ci , C2 
P°p* — „2 + r-4 + •" 


( 21 ) 


and two complex roots that move to infinity as r —> oo. By substituting the series (211 into the 
polynomial equation, we can identify the terms Cj, * = 1,2,.... Thus, we obtain 


Popt — 


2 

j/2 


10 


+ o(r" 4 ). 


4 Conclusion 

In conclusion, for a discrete-time Markov process in the Borel state space with restart, we showed 
that the following statements are equivalent: (a) its expected hitting time to the target set is 
uniformly bounded with respect to the initial states; (b) its expected hitting time to the target set 
is finite for almost all the initial states with respect to the invariant probability; and (c) the target 
set is of positive measure with respect to the invariant probability. In case these statements hold, 
we also established the existence of an optimal restart probability in minimizing the expected 
hitting time to the target set. Finally, two examples were provided to illustrate our results. 
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